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Abstract 


To respond to a user’s query, dialog agents can use a knowl- 
edge base that is either domain specific, commonsense (e.g., 
NELL, Freebase) or a combination of both. The drawback is 
that domain-specific knowledge bases will likely be limited and 
static; commonsense ones are dynamic but contain general in- 
formation found on the web and will be sparse with respect to 
a domain. We address this issue through a system that solicits 
situational information from its users in a domain that provides 
information on events (seminar talks) to augment its knowledge 
base (covering an academic field). We find that this knowledge 
is consistent and useful and that it provides reliable information 
to users. We show that, in comparison to a base system, users 
find that retrievals are more relevant when the system uses its 
informally acquired knowledge to augment their queries. 
Index Terms: Spoken Dialog, Knowledge Base, Situated Do- 
main, Human Knowledge 


1. Introduction 


In many information access applications, systems need to up- 
date their domain knowledge over time to maintain accuracy. 
For example, an event-recommendation agent would need to 
know when new events appear in its domain. General infor- 
mation is available from the web. For example commonsense 
knowledge bases such as NELL [1], Freebase [2]. The agent 
can find information in these knowledge bases, but only about 
popular entities. Other types of information are also available, 
for example about popular concerts, but only those with a web 
presence. We are interested in acquisition of the latter type of 
knowledge, not always present on-line but shared on an infor- 
mal basis within groups, and how it can be obtained through 
interaction with people. 

Knowledge bases such as NELL, Freebase, and Wordnet 
can help in expanding semantic context, thus improve text clas- 
sification in low training data scenarios [3, 4, 5]. In our previous 
work [6], we found that these knowledge bases are useful in im- 
proving dialog task prediction by expanding a user query with 
additional semantic context. However, the semantic context is 
only applicable to common content words as opposed to spe- 
cific entities in the domain e.g., names of specific events. On the 
other hand, people that might interact with a dialog system pro- 
viding information access in a domain, can provide knowledge 
which is useful. The dialog agent could therefore proactively 
seek information from its users and build a knowledge base or a 
folksnomy for its domain. This process is sometimes known as 
collaborative tagging [7]. As a result, the agent gains access to 
ontologies of information that are present in the users’ minds. 

Building knowledge bases is essential for a system that 
needs to answer user queries with situated and contextual in- 
formation. Systems such as Google Now [8], Apple Siri [9] use 
a variety of secondary knowledge sources about the user to an- 


swer a query. This knowledge might include the user’s previous 
search behavior, emails and other information about the user r 
Such knowledge is not readily available in a general common- 
sense knowledge base or a domain knowledge base for a typical 
dialog agent. To this end, we aim to define dialog strategies 
that an agent can use to build up a knowledge base based on 
information obtained through interaction with users. 

Collaborative effort from people can help an agent solve 
complex problems. For example [10] have shown that people 
can build a commonsense knowledge base by playing games 
with a computer. Our work aims at building a knowledge base 
through purposeful dialog between a system and its users. [11] 
have shown that a variety of information can be obtained: for 
example, people can provide answers to visual questions and 
aid physically-disabled users. In our work, we seek to acquire 
knowledge available only through users to help the system pro- 
vide better quality responses to subsequent user queries. We 
address the following questions: 


1. Can dialog-driven acquisition capture domain knowl- 
edge? The agent solicits information from its users ac- 
cording to dialog strategies. This information is used to 
augment a knowledge base for the domain. We evalu- 
ate the coverage provided by this knowledge base both 
qualitatively and quantitatively. 


2. Is the acquired knowledge useful to the system? The 
knowledge acquisition process should aid the system to 
improve its task success rate. To this end, we evaluate the 
system’s performance in its event information-access do- 
main, before and after knowledge acquisition. We show 
that acquired knowledge significantly improves the sys- 
tem’s performance, as assessed by independent judges. 


This paper is organized as follows. First, we describe 
knowledge acquisition strategies and a user study where system 
acquires information from users using these strategies. In Sec- 
tion 3, we describe how agent learns a knowledge base from the 
acquired information. We also present a qualitative and quanti- 
tative analysis of the knowledge base. In Section 4, we evaluate 
the performance of the system on an information-access task, to 
show that acquired knowledge is indeed useful for system per- 
formance. Finally, we make concluding remarks and brief about 
future directions to this work. 


2. Situated Knowledge Acquisition Study 


We posit three different ways in which a dialog agent can inter- 
actively acquire new information from users: 

(I) Knowledge acquired through explicit user guidance. For 
example, a human points at a blue box and tells the agent this 


Inttp://googleblog.blogspot .com/2013/08/ 
just-ask-google-for-your-flights.html 


Table 1: System initiated strategies used by the agent to drive knowledge acquisition 


StrategyType | Strategy Example of System’s Prompt 
Query Driven QueryEvent I know events on campus. What do you want to know? 
QueryPerson I know some of the researchers on campus.Whom do you want to know about? 
: Buzzwords What are some of the popular phrases in your research? 
Egocentric 7 
FamousPeople | Tell me some well-known people in your research area 
Tweet How would you describe this talk in a sentence, say a tweet. 
Show & Ask Keywords Give keywords for this talk in your words. 
People Do you know anyone who might be interested in this talk? 


is a blue box. This form of learning assumes that human is 
an expert user of the system and knows exactly what the agent 
does not know. In a domestic robot domain [12, 13] have shown 
that users can teach novel locations to robots using a combina- 
tion of trigger phrases and non-verbal gestures directed at the 
robot. [14] have demonstrated that system can learn one-word 
descriptions for objects in the environment. It has been shown 
that expert users can teach route plans to a mobile robot [15, 16]. 

(II) Knowledge acquired through “non-understanding” ex- 
changes; i.e., the system cannot extract expected slot-values 
from an utterance [17]. For example, the system detects that 
there is an out-of-vocabulary(OOV) entity in the utterance, then 
tries to clarify with user and learn this entity. [18, 19] have pro- 
posed speak-spell strategy to acquire spellings of OOV words 
through interaction. [20, 21] has shown that a system can detect 
and learn new words in a speech translation application. 

(IID) Knowledge acquired through elicitation. System ini- 
tiates a dialog with user and elicits new information. In their 
Nanoklaus system, [22] used an exploratory strategy to ask 
questions about a concept introduced by the user. In [23], users 
are provided with a saybox to ask a factoid question and the sys- 
tem uses this question to create an agenda of questions probing 
unknown information related to this question. [24] has proposed 
show &ask strategies to elicit new information from people. 

Based on the strategies proposed in [23, 24], we propose 
spoken dialog strategies to elicit information from users. 


2.1. User Study Design 


To investigate how people respond to system initiated knowl- 
edge acquisition, we conducted a user study. The study took 
place in the context of EventSpeak Dialog System that informs 
people about upcoming talks/events of their interest and ongo- 
ing work of other researchers on a university campus. This sys- 
tem takes a spoken query from user and responds with a set 
of talks related to the query. System has a database of 160 aca- 
demic talks that are scheduled between April 2013 to May 2014 
with metadata for each talk: title, speaker name, abstract, loca- 
tion and others. 

To acquire situated knowledge, the agent uses the strate- 
gies shown in Table 1. In Query Driven strategies, system 
prompts user with an open-ended question akin to “How-may- 
I-help-you” to learn what “values” of a slot that user is inter- 
ested in. This allows the system to respond to the query if it 
is familiar with the slot-value, otherwise it learns the unknown 
slot-value. In Egocentric, the system asks user about their own 
interests and people associated with those interests. This allows 
the system to learn information that people are already famil- 
iar with. In Show & Ask, system shows a description of an 
event and asks questions to ground user’s responses in relation 
to that event. This allows the system to learn information in 
relation to an event. This strategy is similar to one strategy pro- 
posed in [24], where the user is expected to provide a title when 


given a story description. Our study is web-based and involves 
recording of voice responses from people based on the system’s 
prompt. We use Testvox’ to display talk descriptions on web 
pages and Wami’, a push-to-talk interface, to record user’s spo- 
ken responses. 

We recruited 40 graduate students from School of Com- 
puter Science at Carnegie Mellon, a representative sample of 
prospective users for EventSpeak system. Each subject per- 
formed spoken tasks involving Query Driven, Egocentric and 
Show & Ask strategies. System initiates a knowledge acquisi- 
tion dialog with the user as illustrated in Table 1 and expects 
the user to respond with a spoken reply. The details of tasks are 
given below. 

In the Query Driven tasks, system responds back to the 
user’s spoken query in the QueryEvent strategy with a list of 
talks. System displays a list of talks if the slot-value in the query 
is familiar, otherwise it does not but learns the new information. 
In the Egocentric tasks, system uses Buzzwords strategy, asking 
the user about popular key-phrases in their research area. Then, 
system asks about well-known researchers (FamousPeople) in 
their area. In the Show & Ask tasks, system shows a description 
of the talk and asks three questions related to this description. 
It asks the user to give one sentence description about the talk 
(Tweet), key-phrases related to the talk (Keywords) and people 
who might be interested in the talk (People). 


2.2. Corpus Collection 


This user study yielded 64 minutes of audio data, with on av- 
erage 1.6 minutes per participant. We have orthographically 
transcribed the user utterances. Then annotated the corpus* for 
people names, and research interests. Table 2 shows the num- 
ber of unique slot-values found in the corpus. We observe that 
Egocentric task yielded relatively higher number of researcher 
names (the FamousPeople strategy) than other tasks. This may 
have happened due to people finding it easier to recall peo- 
ple names from their own research area, compared to names 
in other areas. Overall, the user study yielded 139 unique re- 
searcher names and 485 research interests. 


Table 2: Breakdown of unique number of researcher names and 
researcher interests elicited/acquired by strategy type 


StrategyType | Researcher Names | Research Interests 
Query Driven | 21 29 

Egocentric TI. 107 

Show & Ask 76 390 

Overall 139 485 


2nttps://bitbucket .org/happyalu/testvox 

3https://code.google.com/p/wami-recorder 

“Corpus is available for download http: //www.speech.cs. 
cmu.edu/apappu/pubdl/event speak_corpus.zip 


Figure 1: Condensed version of the block partitioned net- 
work of researchers. Each vertex represents a block of re- 
searchers. 


Figure 2: Research Interests associated with each block in the network. 
Blocks with similar interests have thicker edges. 
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3. Acquired Situated Knowledge Base 


Table 3: Mean Precision for 200 researchers breakdown by the 


In this section, we address our first question: Can the dialog- “source” strategy used to acquire their name. 


driven acquisition capture domain knowledge? From the corpus Source-of-Instance | Researchers | Mean Precision 
collection, we have a list of researcher names and a list of re- Query Driven 21 86.2% 
search interests. To address our question, the system should in- Egocentric 77 93.6% 
fer a list of interests for each researcher i.e, link each researcher Show & Ask 76 86.9% 
to a set of interests. In short, the system creates a bipartite graph Talk Description 61 89.5% 
with links between two disjoint sets: researchers and interests. Overall 200 90.5% 


We quantiatively analyze the consistency of this bipartite graph 
with respect to domain. We analyze this graph qualitatively by 


creating a network of blocks/communities of researchers based 
on their mutual interests. Details given below. 


3.1. Entities and Relations 


We have a disjoint list of entities: (a) researchers and (b) re- 
search interests. Our goal is to infer a list of interests for each 
researcher. For each researcher that was co-mentioned with a 
research interest, we create a link between researcher and that 
interest. For example, in a given dialog session with a user, re- 
searcher names mentioned in FamousPeople strategy are linked 
to interests mentioned in Buzzwords strategy. We repeat this 
process for researcher names and interests mentioned with re- 
spect to a talk i.e., Keywords associated with a particular talk 
are linked to People mentioned with that talk. This process pro- 
duces a bipartite graph with connections between researchers 
and research interests. An example of researcher and predicted 
interests: 


rich stern: deep neural networks, speech recognition, 
signal processing, neural networks, machine learning, 


speech synthesis 


We have 200 researchers (including the ones listed on the 
talk description), each mapped to a subset of interests from 485 
unique interests. On average a researcher has 7.8 interests, with 
a standard deviation of 7.6 (this is because some researchers 
got more mentions across talks than others). We observed that 
some interests are linked to researchers more often than oth- 
ers — machine learning, information retrieval and big data are 
top-3 interests, linked with 49% of the researchers. To assess 


the quality of predicted interests, we asked two senior Carnegie 
Mellon graduate students to label whether a predicted interest of 
a researcher is accurate. Table 3 shows the mean precision ° for 
the predicted interests with a breakdown by source of researcher 
name and has good accuracy irrespective of the source. 

To better understand how researchers are linked to interests 
and in general how researchers are aligned to each other, we 
construct an adjacency matrix of researchers. The details are 
described in the next subsection. 


3.2. Analysis of Entity Network 


To create an adjacency matrix of entities, i.e., the researchers, 
we compute the Jaccard index [25] for each pair of researchers 
based on their interests. The Jaccard index compares similarity 
of sample sets (say A and B) and is computed as: 


ANB 
J(A, B) = sor a (1) 


We use the Jaccard index of two researchers as the weight of 
their connecting edge. We then convert the adjacency matrix to 
a network (an undirected graph) using a graph tool package’. 
To find communities or blocks in the resultant network, we use 
a stochastic block inference algorithm [26]. This algorithm tries 
to minimize description length (MDL) of the network (measured 
in nats-per-edge) to produce a block-partitioned version of the 
network. Intuitively, a block represent a set of nodes that more 


5We use only precision because we do not have exhaustive list of 
relevant interests to measure the recall 
Shttp://graph-tool.skewed.de 


often interact within each other than with rest of the network; 
in our case, blocks are research communities. An illustration of 
a condensed version’ of the network is shown in Figure 1 with 
research interests associated with each block shown in Figure 2. 
Since the block inference algorithm optimizes the MDL, it 
may overfit the number of blocks needed to represent the net- 
work. Although in certain graphs (e.g., citation network) it is 
impossible to have modular or well-separated blocks, it is de- 
sirable to have reasonably separated blocks. Newman’s modu- 
larity [27] is a metric, typically used to measure the strength of 
division in a graph °. To achieve a reasonable separation, we 
ran several iterations of the block inference algorithm, varying 
the minimum number of blocks required for the network. In 
Figure 3, we see that MDL is lowest (8.8) and MODULARITY is 
highest (0.08) when we set the minimum number of blocks to 
16. This yielded a network of 17 blocks, as shown in Figure 1. 


Figure 3: MDL and MODULARITY of network against min. number of 
blocks at initialization 
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4. Using Knowledge: Query Expansion 


Now, we want to know whether the acquired knowledge is use- 
ful to the system This can be demonstrated by showing that the 
EventSpeak System can respond more precisely to user queries 
by returning more relevant talks, after knowledge acquisition 
than before. Previous work [28] has shown that query expan- 
sion based on semantic networks can improve retrieval perfor- 
mance. We compare performance by expanding user query with 
acquired knowledge against the unexpanded queries. 


4.1. Query Expansion Setup 


We built a database of 160 research talks using abstracts, ti- 
tles and speaker names as indices. For this purpose, we use 
Latent Semantic Indexing method as implemented in the GEN- 
SIM toolkit [29]. In the user study described earlier, we col- 
lected 40 queries (29 unique) about research areas and 40 (21 
unique) about researchers — a total of 50 unique queries. We 
use these 50 queries to evaluate retrieval performance. Research 
area queries were expanded with their top-3 co-occurring re- 
search interests in the corpus. For researcher queries, we used 
the top-3 researcher’s interests. Ten results with and without 
query expansion were retrieved. We then asked human judges 
(senior graduate students) to assess the relevance of each result 
on a scale of 1-4 (higher the better) with respect to a query. Our 
hypothesis is that results based on query expansion will have 


7full-blown network is attached in the submission 
8Modularity cannot capture blocks/communities in smaller 
graphs/networks, hence we do not use it directly for block partition 


higher relevance compared to the results without query expan- 
sion. 


4.2. Query Expansion Results 


We asked 5 human judges to rate the relevance of results from 
50 queries with and without query expansion. We measured 
inter-annotator agreement by having a pair of judges annotate 
the same set of 20 queries. Overall we obtained ratings for 100 
(5x10x2) queries: 5 sets of 10 queries each, rated by 2 judges 
per set — with good agreement (Cohen’s & = 0.41). 


Table 4 shows the mean relevance, as rated by the annota- 
tors, for the retrieved talks with respect to the query. The query 
based expansion system outperforms the baseline, as observed 
in previous work [28]. Expansion works particularly well for 
“researcher” queries. One reason may be that person names 
may not have appeared in the talk description (and not indexed), 
but the research interests used to expand the query may appear 
in the talk descriptions (and indexed). We show that using this 
knowledge in expanding user queries can result in significantly? 
(p < 0.01) more relevant results (2.5/4 vs 1.8/4) than before ac- 
quisition. 


Table 4: Mean relevance-per-query on scale of 1-4 (higher the better). 
Knowledge-based query expansion results are statistically (p < 0.01) 
more relevant than those without expansion. 

QueryType Without Expansion | With Expansion 
Researcher 1.1 (stdev=0.8) 2.4 (stdev=0.6) 
ResearchArea | 2.2 (stdev=0.6) 2.5 (stdev=0.6) 
Overall 1.8 (stdev=0.9) 2.5 (stdev=0.6) 


5. Conclusion and Future Work 


We describe an agent that learns domain knowledge through di- 
alog with users and uses it to build a semantic representation of 
an academic field. The system uses a set of strategies to col- 
lect entities (researchers and research interests). These entities 
are linked by their co-occurrence to produce a bipartite graph 
linking researchers and research interests. 


To verify that the acquired knowledge is consistent, we 
asked human annotators to judge whether the interests predicted 
by the system were accurate. We found that the predicted inter- 
ests for researchers have a high mean precision of 90.5%, i.e., 
annotators agree with the system’s predictions in most cases. 
To analyze this knowledge qualitatively, we build a network 
of researchers connected through their mutual interests and di- 
vide this network into blocks using a block inference algorithm. 
This results in a set of blocks/communities of researchers (like 
a citation network) that covers the original academic field. We 
found that acquired knowledge used for query expansion pro- 
vides more relevant results (2.5/4 vs 1.8/4), according to human 
judges, than without this acquired knowledge. 


Knowledge acquisition involves cost (user’s time) and in- 
centive (new knowledge for system). Designing a policy to as- 
sess cost and incentive of acquisition, will be an immediate goal 
for this work. This policy depends on number of factors: iden- 
tify the worth of unknown information, judge user’s expertise 
about unknown information, etc. A policy with such factors 
may guide the system when to learn and what to learn. 
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